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1. Background 

In the Spring of 1991, Defense Contract Management Command selected the Defense Plant 
Representative Office (DPRO) at TRW Space & Defense as one of seven organizations nationwide 
to develop and implement a Performance Based Management (PBM ) Plan — a Total Quality 
Management (TQM) initiative. DPRO elected to develop the PBM Plan with TRW and to pursue it 
jointly. TRW and DPRO concurred on the Plan and mutually agreed on implementation. 


2. Introduction 

As the nation redefines priorities to deal with a rapidly changing world order, both government and 
industry require new approaches for oversight of management systems, particularly for high 
technology products. Declining defense budgets will lead to significant reductions in government 
contract management personnel. Concurrently, defense contractors are reducing administrative and 
overhead staffing to control costs. These combined pressures require bold approaches for the 
oversight of management systems. 

In the Spring of 1991, the DPRO and TRW created a Process Action Team (PAT) to jointly 
prepare a Performance Based Management (PBM) system titled Teamwork for Oversight of 
; Processes and Systems (TOPS). The primary goal is implementation of a performance based 
management system based on objective data to review critical TRW processes with an emphasis 
on continuous improvement. The processes are: Finance and Business Systems, Engineering and 
Manufacturing Systems, Quality Assurance, and Software Systems. The team established a 
number of goals. 

• Delivery of quality products to contractual terms and conditions, 

• Ensure that TRW management systems meet government guidance and good business 
practices, 

• Use of objective data to measure critical processes^ 

• Elimination of wasteful/duplicative reviews and audits, 

• Emphasis on teamwork — all efforts must be perceived to add value by both sides and 
decisions are made by consensus; and 

• Synergy and the creation of a strong working trust between TRW and the DPRO. 

TOPS permits the adjustment of oversight resources when conditions change or when TRW 
systems performance indicate either an increase or decrease in surveillance is appropriate. Monthly 
Contractor Performance Assessments (CPA) are derived from a summary of supporting system- 
level and process-level ratings obtained from objective process-level data. Tiered, objective, data- 
driven metrics are highly successful in achieving a cooperative and effective method of measuring 
performance. 

The teamwork-based culture developed by TOPS proved an unequaled success in removing 
adversarial relationships and creating an atmosphere of continuous improvement in quality 
processes at TRW. The new working relationship does not decrease the responsibility or authority 
of the DPRO to ensure contract compliance and it permits both parties to work more effectively to 
improve total quality and reduce cost. By emphasizing teamwork in developing a stronger 
approach to efficient management of the defense industrial base TOPS is a singular success. 


3. Responsibilities 

TOPS consists of four main areas: Finance and Business Systems, Quality Assurance, 

Engineering and Manufacturing, and Software, 

3.1 Finance and Business Systems 

Finance and Business Systems covers the following TRW functions: contracts, pricing and cost 
data systems (estimating system), procurement, and property. Three of the systems, (pricing and 
cost data systems [estimating system], procurement, and property) require written certification by 
the Divisional Administrative Contracting Officer (DACO) that TRW policies and procedures 
satisfy all applicable laws, rules, and regulations of the government. The program applies to all 
TRW organizations in the Los Angeles area under DPRO cognizance. 

3.1.1 TRW Functional Organization for Finance and Business Systems 

3. 1.1.1 Contracts 

The contracts function participates in the sales acquisition process; serves as the official channel for 
business communications with the customer; conducts negotiations and commits TRW; determines 
contract requirements; and provides direction to company organizations regarding contractual 
compliance. Contracts also protects TRW financial, legal, ethical, and proprietary interests while 
promoting compliance with applicable laws and acquisition regulations. 

3. 1 . 1 .2 Pricing and Cost Data Systems (Estimating System) 

Pricing and Cost Data Systems provide financial leadership during the strategy, planning, 
development, and support of cost volumes. They strive to meet or exceed the needs of external 
and internal customers during cost estimating and pricing activities. Pricing and Cost Data Systems 
assures compliance with the Truth in Negotiations Act, tne Estimating System Disclosure^ . 
Statement, and all applicable laws and regulations. An element of the Estimating System is indirect 
and direct labor rate management activities. These functions are normally performed within the 
controller's organization of the applicable profit center. 

3. 1.1. 3 Procurement (Subcontracts and Purchasing) 

Sole authority for selecting sources; obtaining quotations; and awarding orders for materials, 
products, equipment, and services lies with Procurement, conducting these activities in the most 
economical and efficient manner. To accomplish this mission, purchasing serves as the official 
channel of communication to all suppliers; acts as negotiation agent of the company; and ensures 
compliance with all legal, ethical, administrative and documentation policies. Subcontracts 
manages acquisition from identification of need, to obtaining proposals or quotes, through final 
delivery and acceptance of systems, equipment, products, material, and services. They ensure 
compliance with all quality, reliability, technical, legal, and administrative requirements. 

3. 1.1. 4 Property 

A written Property control system provides for proper management and use of corporate assets and 
government property. Contracts require TRW to meet specific minimum requirements for the 
control, protection, preservation, and maintenance of all government property accountable to those 
contracts. 
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3 . 1 . 1 . 5 Critical Processes 

Both TRW and the DPRO will review each functional system for the selection of "critical 
processes" for measurement. These reviews ensure compliance with government policy and verify 
efficient and effective operations. Some critical process measures reflect DPRO performance, such 
as the number of days it takes the DPRO to obtain a completed assist field pricing audit following a 
request from TRW. Criticality, impact, opportunity for improvement, and the ability to maintain 
regular surveillance will govern process selection. Following this review, TRW and the DPRO 
will agree on areas to measure. If a disagreement occurs, the DPRO position takes precedence. 
Addendum A lists current critical processes. 

3.1.2 Finance and Business Systems Evaluation 

To remove as much subjectivity as possible the performance of the critical processes will be 
measured with statistical process control (SPQ charts. The rating methodology of each critical 
process includes four key criteria: existence of process data (metrics), acceptability of the process, 
state of process control, and measurable improvement. 

Existence of Data. The TRW process owner/evaluator has available an acceptable process 
performance metric and supporting data of the critical process. 

Acceptability. Meeting statutory, regulatory; or contractual requirements. Processes whose 
output cannot be traced to statutory, regulatory, or contractual requirements will have their level 
of acceptability determined by the process owner/evaluator with the concurrences of the 
Finance and Business subteam. 

Control A measure of the deviations of the SPC data through the upper control level (and 
lower control level when appropriate for things such as rates), its variability, or trends that 
represent control of the process. 

Improvement. The last eight data points show a positive trend, a reduction in variability, or 
have reached a level below which a further reduction would not be cost effective or meaningful 
(to be determined by the process owner/evaluator with the concurrences of the Finance and 
Business subteam). 

Evaluators analyze the data collected on SPC charts and complete a rating of the four key processes 
with a point assignment as follows: 0 or 0.5 for existence of data and for improvement and 0, 0.5, 
1.0, or 1.5 for acceptability and for control. Ratings of the critical processes in a functional area 
are weighted (if appropriate) and averaged to provide a monthly Contractor Performance 
Assessment (CPA). Ratings are prepared by the process owner/evaluator and presented with the 
SPC charts for concurrence at the monthly Finance and Business subteam meeting. The 
consolidated Finance and Business CPA is the average of the four functional CPAs. 

3.2 Engineering and Manufacturing 

3.2.1 Identification of Top-Level and Critical Processes 

The Engineering and Manufacturing subteam based the identification of top-level and critical 
processes on the product development life cycle because it describes, in a generic manner, the steps 
a product goes through from conception to delivery. Process definitions fall into one of two 
categories: generic, top-level processes that make up the product development life cycle, or 
processes critical to a generic process that ensure an aspect of success. In the selection of top-level 
processes, TOPS uses brainstorming techniques and achieves consensus for each of the seven 
processes defined between concept definition and satellite delivery. They are: systems 
engineering, detailed design, subcontracts, parts procurement, manufacture, support equipment, 
and systems integration and test. Figure 1 shows a generic flow of the processes. 
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Figure 1. Generic Process Flow 

The subteam chose critical processes from a high-level flowchart of each top-level process, as 
depicted in Figure 2. 
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Figure 2. Top-Level Process Flow 









To ensure correct selection of critical processes and to facilitate meaningful metric selection, the 
subteam completes a process schematic for each critical process. A process schematic defines the 
inputs, outputs, customers), customer expectations, quality expectations, and possible 
measurements (Figure 3). The document "Engineering & Manufacturing Subteam Process 
Package" contains high-level flowcharts for the top-level processes and process schematics for 
each critical process. 
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Figure 3. Unit Specification Generation Process 
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3.2.2 Identification of Process Performance Metrics 

To remove as much subjectivity as possible from the rating scheme, the subteam, with the aid of 
the process owners, identifies process performance metrics (which characterize quality and cycle 
time) for each critical process, using the process schematic. The subteam avoids "reinventing the 
wheel" wherever possible using applicable metrics previously defined by TRW personnel in 
company TQM activities such as the Design Through Manufacturing (DTM) and Satellite Cost 
Reduction (SCORE) efforts. Defined metrics can and will change when new or superior metrics 
are established, or deleted if found to provide no added value. 

3.2.3 Methods of Process Performance Analysis 

To facilitate analysis statistical methods, preferably the use of SPC charts, will determine process 
trends; however, processes unsuitable for the application of SPC charts use other statistical means 
for interpreting data. 

3.2.4 Methods Evaluation and Rating 

3.2.4. 1 Evaluation Criteria 

To give both TRW and DPRO management a clear picture of the health of critical processes 
throughout TRW, the rating methodology embodies four key criteria: existence of process data 
(metrics), acceptability of the process, state of process control, and measurable improvement. 
Analysis of the collected metrics, plus other tangible and intangible information, determine point 
assignments. Where applicable, evaluators will normalize and analyze data via SPC charts, with 
point assignment as follows: 1 point for existence of data, 1 point for acceptability, 1 point for 
being in-control, and 1 point for improvement. Points for each critical process are equally 
distributed among the two categories of measurements (cycle time and quality). If tjfefe Sf e 
multiple quality measurements, then points within the quality category are equally distributed, as 
well. 

Key Term Descriptions: 

Existence of Data. Initially, it is defined as performance metric(s) identified by the process 
owners and concurred by the subteam to be meaningful. The definition will change as the 
development of metrics mature. 

Acceptability. 1) One or two performance metrics that measure a significant element of the 
process (such as span time, efficiency, or quality of the process). 2) Additional metrics must 
be developed and implemented in a reasonable time frame. 3) The absolute value of metric (for 
example, 100 EO's per drawing) is also considered as applicable. 

In-Control. 1) The measured data have established control limits or are judged by the process 
owner to be in control, but eventually must have established control limits. 2) If variability 
swings widely, process owner must explain. 

Improvement. A positive trend is either a positive shift in the process mean or a reduction in 
variability. 

3. 2.4. 2 Evaluation Method 

Evaluation teams evaluate and determine a grade for each critical/top-level process and give the 
result first to the process owner, then to the TOPS Engineering and Manufacturing subteam for 
incorporation into the overall group evaluation. The final grade, along with any necessary back-up 
documentation, goes to the DPRO Commander and the appropriate TRW Vice-President/General 
Manager (VP/GM) as part of the comprehensive TOPS rating. This structure pushes 
responsibility, accountability, and evaluation down to the level of work accomplishment. The 
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process owner knows the grade for his or her process(es) before the grade reaches upper 
management, enabling timely response preparation. 

Figure 4 illustrates this method of process performance assessment. 


Top-Level or Critical Process 



Figure 4. Top-Level or Critical Process Performance Assessment 
Key Term Descriptions. 

Assessment Grade. That grade determined from the evaluation criteria (see Section 3.2.4.1). 

Discussions. Interactions with the process owner - interviews, investigations of out-of-control 
conditions, joint reviews, etc. 

DPRO Evaluator. A DPRO employee knowledgeable of the TRW process. 

Evaluation Team. A DPRO evaluator and a TRW evaluator who review the process 
performance metrics, discuss/investigate issues with the process owner, and determine an 
assessment grade. 

Process Owner. The person responsible/accountable for process performance. 

TRW Evaluator. A TRW employee knowledgeable of the process under evaluation and neither 
responsible for nor associated with the process performance. 

Periodic evaluation of top-level and critical process occur as defined later in this guide. 
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3.3 Quality Assurance 

Under TOPS, DPRO and TRW Space & Technology Group (S&TG) and Electronic Systems 
Group (ESG) will jointly review TRW quality systems and functions to support and facilitate the 
DPRO Quality Assurance (QA) Division's implementation of In-plant Quality Evaluation (IQUE). 
For Systems Integration Group (SIG) West, periodic audits are performed in lieu of TOPS 
surveillance. The DPRO/QA and TRW/QA group-level Major Support Processes (MSPs) are 
those key quality processes jointly identified by DPRO/QA and TRW/QA. Key subprocesses 
within each MSP are identified, flowcharted, and proofed, with measurement points selected or 
metrics developed for trending, analysis, and evaluation by the TO PS QA sub committee and 
DPRO and TRW senior management. Management takes action as necessary to correct or to 
prevent undesirable trends or to address major deficiencies, problems, or concerns. 

3.3.1 Responsibilities 

As an integral part of the overall TOPS effort, a joint DPRO/TRW QA Steering Committee (TOPS 
QA subco mmi ttee) meets regularly to direct and assure effective implementation of the TOPS QA 
efforts in support of the PBM initiative. This team will create and empower ad hoc working teams 
of DPRO/QA Operations Support Branch (RTQT) and TRW/QA personnel to analyze processes 
and data, to develop metrics and audit techniques, to work special problem areas, and to assist in 
implementation of the TOPS QA efforts. TOPS QA ratings, i.e., "T OPS-le vel metrics," will be 
prepared jointly and assessment reports made to DPRO and TRW management by the TOPS QA 
subcommittee. The TOPS QA subcommittee will evaluate the progress, problems, and lessons 
learned and redirect the overall effort as appropriate. 

3 . 3 . 1 . 1 Mission Statement 

Identify and define major support processes, emphasize the objective of continuous improvement, 
jointly, through consensus, develop objective criteria for an overall assessment method for 
measurement of the processes. 

3.3.1 .2 Goals and Objectives 

a. Flow key quality processes and define associated value added metrics. 

b. Reduce oversight through: 

(1) Achievement of confidence in key quality processes. 

(2) Institutionalizing process metrics for major support processes (MSPs). 

(3) Building trust between DPRO and TRW. 

(4) Conducting joint reviews. 

3.3.2 Process Identification, Proofing, and Monitoring 

3 . 3 . 2. 1 Major Support Processes 

Addendum C identifies the nine MSPs which describe the key quality processes. DPRO/QA and 
TRW/QA will use these MSPs to monitor the performance of TRW’s quality system. 

3.3. 2.2 Process Selection 

The TOPS QA team jointly selects the subprocesses in each MSP based on: 

a. Identification by the Government and/or TRW. 

b. Analysis of data which reflects opportunities for improvement. 

c. Process(es) which represent the key quality systems. 


F3.1-9 


3. 3. 2.3 Process Proofing and Monitoring 

The ad hoc teams, appointed by the TOPS QA team, perform the following functions: 

a. Flowchart the "as is" process(es). 

b. Proof (validate) the process(es) against policy, procedures, and contract requirements. 

c. Recommend metrics to measure the processes). 

d. Identify measurement points. 

e. Perform process and/or product audits. 

f . Collect, compile, review, trend, and analyze data and develop trend charts. 

g. Use statistical process control (SPC) charts, as applicable. 

h. Provide recommendations for process improvements. 

3. 3. 2. 4 Continuous Improvement Opportunities (CIOs) 

Continuous Improvement Opportunities (CIOs) normally found when analyzing contractor data, 
conducting product audits, or proofing the adequacy of a process are submitted to the contractor. 
Implementation of CIOs by the contractor is optional. However, the contractor upon request by 
the government will provide feedback whether improvements were implemented or not. The TOPS 
QA team uses the standard format methods described below to issue CARs. 

3.3.3 Standard Format Methods 

3.3.3. 1 Corrective Action Requests (CARs) 

CARs consist of two types: verbal or written (Internal Compliance Review Reports [ICRRs]). 
Determination of which type to issue depends on the following factors: criticality of the 
nonconformance; frequency; effect on reliability, maintainability, or operability; whether the 
contractor has planned or taken corrective action (C/A); reluctance to initiate C/A; effectiveness (or 
lack thereof) of previous C/A; etc. 

In all cases, DPRO/QA will: discuss the CAR with the responsible element(s) of both the 
contractor's performing organization and the TRW quality organization. Distribute written CARs 
(ICRRs) to the responsible discipline(s) and the quality organization. Enter all CARs (verbal and 
written) into a database. Verify adequacy of the C/A proposed, initiated, and implemented by the 
contractor, as evidenced by the absence or reduction of the defect in follow-on analysis of data or 
process/product audits. Record this verification in a log or record (manual or computer based). 

3.3.3. 2 Method C 

Chronic or systemic process problems not addressed by the contractor typically indicates a failure 
to recognize that the problem exists. Normally, DPRO/QA issues a letter of concern requesting 
C/A. Inadequate C/A results in the issuance of a Method C. When serious quality problems exist 
and the contractor fails to take positive C/A, a letter forwarded to senior contractor management 
through the TRW DPRO Liaison Office will request immediate C/A for the observed deficiencies 
and their causes. 
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3. 3.3. 3 Method D 

Where the contractor cannot or will not comply with contract requirements and C/A cannot be 
effected directly with the contractor by other methods, DPRO/QA personnel will request that the 
Administrative Contracting Officer (ACO) inform the contractor that all IQUE actions will be 
discontinued as required by Defense Logistics Manual DLAM 8105.1, "Contract Administration 
Manual for Contract Administration Services." This initiates Metho d D e s calat ion. The ACO, in 
consultation with DPRO/QA personnel, determine the appropriate course of action. Government 
IQUE actions will totally discontinue only when authorized in writing by the ACO. 

3.3.3.4 Method E 

Where a subcontract is involved and the requirements for C/A are of the magnitude of a Method C 
or D, DPRO/QA will request that TRW take immediate C/A with the subcontractor. 

3.3.4 Quality Assurance Evaluation 

Each MSP rating uses the numerical application listed below, ranging from 0 (unsatisfactory) to 4 
(exemplary). ~ 

4 = Exemplary 
3 = Excellent 
2 = Satisfactory 
1 = Marginal 
0 = Unsatisfactory 

Each TRW group will be rated monthly by MSP (example below): 


Systems 

ESG 

S&TG 

QA1 

4 

3 

QA2 

3 

3 

QA3 

4 

3 

QA5 

4 

3 

QA6 

4 

3 

QA7 

4 

4 

QA8 

4 

4 

QA9 

3 

3 

QA10 

3 

3 

TOTAL 

33 

29 

Adding the 9 elements and dividing by the number of MSPs yields an overall numerical rating for 
each group. The accompanying descriptive rating comes from the following range chart. 

Divide by 9 MSPs 3.6 (Excellent) 

3.2 (Excellent) 

0.0 - 0.5 = 

Unsatisfactory 


0.6 - 1.5 = 

Marginal set 


1.6 - 2.5 = 

Satisfactory 


2.6 - 3.5 = 

Excellent 


3.6 - 4.0 = 

Exemplary 


QA4 Software 

- Assessed by Software Process Improvement Initiative (SPII). 
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3.4 Software 

The software subteam's goal is to develop a system for evaluation of the software development 
processes used at TRW. Once established, the system will be used as a base for continuous 
process improvement initiatives for TRW and for providing an objective measuring system for the 
DPRO. The system will identify the problem areas (or areas which could be improved) and provide 
the ability to detect improvements (or degradings) in the processes. The evaluation process is 
established by identifying metrics and then using continuous process improvement (in terms of the 
validity of the evaluation metrics) to continually improve the measurement system. Two divisions 
will be used to develop the first rating systems. These systems will be analyzed for similarities and 
used as a base for expanding surveillance to the rest of TRW, The similarities in the first two 
divisions and those in the rest of TRW should give some indicators as to the value of different 
metrics (i.e., identical metrics used throughout all the divisions shows a high value of the metric). 

3.4.1 Scope 

Software activities from all Space Park Divisions/Groups as well as the Division at Dominguez 
Hills will be included in these evaluations. At first, evaluations are limited to the SDD and SEDD 
divisions of the Systems Integration Group (SIG). The two divisions will be used in the 
development of evaluation processes and the descriptions contained herein are based on the 
similarities of the two divisions. 

3.4.2 Structure 

The overall TOPS software evaluation has three components: 1) the underlying process capability 
(20% of the grade), 2) the effectiveness of management's implementation of the processes (40% of 
the grade), and 3) the ultimate performance of the processes (40% of the grade). These 
components along with "Key Process Areas" are shown in Figure 5. 

The process capability component of the evaluation is based on four processes. The first depends 
on meeting the maturity improvement goals of the Software Engineering Institute's (SEI) 
Capability Maturity Model (CMM). The SEI was set up by the government (because of the 
increasing life cycle importance of software) to identify and solve problems with software 
acquisition. The CMM provides as a baseline to evaluate the contractors ability to produce the 
required software product within all the procuring agency's constraints. The second and third 
processes involve the use of technology (i.e., introduction and dissemination) within an 
organization. The fourth process for the capability evaluation is based on TQM/CPI practices. 

The management component of the evaluation examines management oversight, project metrics, 
and feedback from the customer. The weightings assigned to each of these categories varies 
depending on the division and/or group. The management oversight entails C/SCSC, CSSR, or 
some other type (if used) of cost/schedule reporting. Project metrics will vary from division (or 
group) to division (or group). Some examples of these are number of source lines of code 
(SLOCs) design and built versus the number planned to be built, number of SLOCs turned over 
(for test) versus the number of SLOCs planned to be turned over, and number of capabilities 
integrated verses number of capabilities planned to be integrated. The actual metrics used depends 
on the division/group as well as the definitions of the measurable quantities. 

The performance component of the evaluation is based on a combination of DPRO and TRW 
quality data. DPRO data is comprised of IQUE activities (e.g., SDF code inspection. The SDF 
code inspections are a metric which identifies the number of SDFs evaluated for coding versus 
SDFs found with code nonconformances.) TRW data is comprised of various contractor 
evaluations/functions conducted by program level QA (contractual) and independent management 
level (system) evaluations. Two types of findings are generated during these reviews - internal 
corrective actions that require formal review and correction and noted recommendations identified 
to the project by the review team. 
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3.4.3 Review Methodology 

For the software process: 

• Senior management within a TRW surveillance area identifies functional points of contact 
(engineering, quality assurance, program management, etc.). 

• These personnel work with the identified DPRO members to establish a joint team to 
identify, prioritize, and implement the processes for surveillance. 

The team shall: 

• Identify the data which will be collected. 

• Collect and report on the agreed upon data. ^ ‘ " 

• Verify the collected data is correct. 

• Interview cognizant personnel, as required. 

• Review documentation, as required (i.e., UDFs, SDFs, etc.) 

• Establish and update the rating (and reporting) system, as required. 

• Identify, evaluate, and implement (as required) capability improvements to the rating (and 
reporting) process. 

• Identify, evaluate, and implement (as required) capability improvements to the software 
development processes. 

Figure 5 illustrates this structure: 



• Evaluations 

• Technology Improvements 

• Technology Dissemination 

• TQM/CPI 


• Management Oversight 

• Project Metrics 

• Customer Assessments 

• Milestone Results 

• Documentation Status 


• IQUE Reviews 

• TRW Compliance 

Reviews 

• Customer Reviews 


Figure 5. TOPS Software Evaluation Components 


F3.1-13 






4. TOPS Evaluation 

Figure 6 presents the flow of rating data within the DPRO and TRW. Each functional subteam 
must supply data based on a five point scale for review by TOPS management at a monthly 
meeting. The lead DPRO representative of each subteam will present the consolidated process 
ratings as an overall group rating (see Figure 7 for a sample group-level assessment chart). 
Additionally, the subteam leader must present back-up data, as appropriate, for concerns that the 
subteam wishes to elevate. Also at the monthly meetings, DPRO and TRW management will 
review and discuss the results of joint or independent evaluations and any other pertinent 
information, including TRW TQM activities. Finally, discussions would cover potential problem 
resolutions and recommended courses of action. These could run from continued problem 
monitoring, to establishing a process action team (joint or independent), to a letter from the DPRO 
to correct the problem. When the DPRO notifies TRW in writing of a problem, the notification 
should go to the TRW DPRO Liaison Office to ensure proper distribution and coordination within 
TRW. Figure 8 presents the corrective action matrix, depicting the methods of notification and 
escalation for each subteam. 




Process Data Collection & Analysis, w/Feedback to Process Owner 


Figure 6. Management/Rating Assessment Structure 
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Figure 7. TOPS Monthly Assessment (Sample Data) 
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Figure 8. TOPS Corrective Action Matrix 
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5. Acronyms 


AGO 

Administrative Contracting Officer 

AFMC 

Air Force Material Command 

APMA 

Automated Property Movement Authorization 

BOE 

Basis of Estimate 

OA 

Corrective Action 

CACO 

Corporate Administrative Contracting Officer 

CAR 

Corrective Action Request 

CAS 

Contract Administration Service 

CCN 

Contract Change Notice 

CER 

Cost Estimating Relationship 

ao 

Contractor Improvement Opportunity 

CPAR 

Contractor Performance Assessment Report 

CPI 

Continuous Process Improvement 

CPSR 

Contractor Purchasing Systems Review 

CSE 

Contractor System Element 

DACO 

Division Administrative Contracting Officer 

DCAA 

Defense Contract Audit Agency 

DCMC 

Defense Contract Management Command 

DCMDW 

Defense Contract Management District West 

DFAR 

DOD supplement to the FAR 

DLA 

Defense Logistics Agency 

DLAM 

DLA Manual 

DOD 

Department of Defense 

DPRO 

Defense Plant Representative Office 

DTM 

Design through Manufacturing 

ESG 

TRW Electronic Systems Group 

FAR 

Federal Acquisition Regulation 

GAO 

General Accounting Office 

I&T 

Integration & Test 

ICRR 

Internal Compliance Review Report 

IG 

Inspector General 

IQUE 

In-Plant Quality Evaluation 

LAN 

Local Area Network 

MGI 

Mandatory Government Inspection 

MOA 

Memorandum of Agreement 

MSP 

Major Support Process 

NASA 

National Aeronautics and Space Administration 

ODC 

Other Direct Cost 

OFPP 

Office of Federal Procurement Policy 

OMB 

Office of Management & Budget 

P&TS 

Program & Technical Support 

PAT 

Process Action Team 

PBM 

Performance Based Management 

PCO 

Procuring Contracting Officer 

PO 

Purchase Order 

PR 

Purchase Requisition 

PROCM 

Procurement Manual 

QA 

Quality Assurance 

R&D 

Research & Development 

RFP 

Request for Proposal 

ROAE 

Return on Assets Employed 

S&D 

TRW Space & Defense Sector 


F3.1-16 



S&TG 

TRW Space & Technology Group 

SCORE 

Satellite Cost Reduction 

SEC 

Securities & Exchange Commission 

SIG 

TRW Systems Integration Group 

SOW 

Statement of Work 

SPC 

Statistical Process Control 

SPII 

Software Process Improvement Initiative 

TBD 

To Be Determined 

TINA 

Truth in Negotiations Act 

TOPS 

Teamwork for Oversight of Processes and System 

TQM 

Total Quality Management 



6. Addenda 


A. Finance and Business Systems 

A.1 Critical Process Oversight 

The DPRO and TRW jointly selected as critical processes those processes essential to DPRO's 
oversight responsibilities and operational effectiveness. Compilation of specific data by TRW will 
facilitate oversight of the critical processes and provide the DPRO with tangible evidence of the 
effectiveness of these processes. DPRO may also select and gather data on some critical processes 
to assess their own performance, e.g., the number of days required for the DPRO to obtain a 
completed field pricing audit report per a TRW request. The DPRO will use statistical process 
control (SPC) charts to assess these processes. SPC charts will be prepared such that situations or 
occurrences that would be deemed to be "out of tolerance" by the DPRO will be easily identifiable, 
so the measurement of the number of such occurrences is clear and indisputable. 

A.2 Critical Processes 

A. 2.1 Contracts 

A.2. 1 . 1 Contract Closeout 

After contract completion, when the product and/or service is delivered to and accepted by the 
customer, a specific closeout process is conducted. Before TRW submits a final invoice to the 
customer, several internal closeout actions are completed as specified in Contracts Manual section 
501, "Closeout of Cost Type Contracts." On completion of these actions, a final invoice, release, 
and assignment are prepared. For this critical process, the time intervals identified below will be 
tracked and measured. The source of the data points will be the Contract Closeout Status Report 
CM071EM published by the Contract Information Center. The data tracks the time interval 
between the date the contract entered into closeout and the date all closeout actions are completed. 
The time interval between the date the final invoice is submitted to the DPRO and the date payment 
is received from the paying office is also tracked.. 

A. 2 . 1 . 2 Restrictive Markings Notification 

Contracts manual bulletin number 16 requires that the contract administrator to provide written 
notification to the contracting officer within 60 days after award of the name and title of the person 
having final responsibility for determining whether restrictive markings are to be placed on 
technical data to be delivered under the contract. In order to monitor TRW’s compliance with this 
clause, we will track the time interval between the date of contract award and the date restrictive 
markings notification is submitted by the contract administrator. 

A. 2.1. 3 Waivers and Deviations 

When project management determines that a need exists to request a waiver to or deviation from 
contractual specifications, the waiver/deviation is prepared in a manner suitable for government 
review and approval. Waivers and deviations are granted by the cognizant government contracting 
officer. For this critical process, the following time intervals will be tracked and measured. The 
time interval between the date the waiver or deviation was submitted and the date the approval or 
denial was received from the DPRO/customer. 

A.2. 1.4 Progress Payments 

Progress payments will be tracked from the time they are prepared by Accounting Operations, 
delivered to the DPRO or DCAA, as appropriate, and returned to Accounting Operations after 
approval by the cognizant government office. 
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A.2.1.5 Invoices 

Invoices will be tracked from the time they are prepared by Accounting Operations, delivered to the 
DPRO or DCAA, as appropriate, and returned to Accounting Operations after approval by the 

cognizant government office. 

A.2.1.6 Cost Accounting Standards Disclosure Statements 

The Cost Accounting Standards (CAS) require TRW to submit disclosure statements regarding its 
accounting practices. These disclosure statements are amended each time a change m cost 
accountin! practices occurs. TRW must submit amendments to the govemmwt W d^s pnor to 
their effective date. As a process essential to the government s oversight responsibility, TOPS will 
mej^urethe^ adequacy and timeliness of TRWs CAS disclosure statement modifications. 

A 2.2 Pricing and Cost Data System (Estimating System) 

Six processes have been identified as critical in the Pricing and Cost Dahl i S d 
Svstenri area The primary areas of focus were identified as part of the joint DCAA, DPRO, ana 
TOW Proposal Process Sion Team (PAT), and the recently completed Estimating System Survey 
conductecfby DCAA and the DPRO. In addition the DACO must submit wntten certification that 
the TOW Estimating System satisfies all applicable laws, rules, and regulations of the govemmen . 
To ei^ure thauhe ef. bating system requirements are contmually m*g*c P™ * lU be 
monitored monthly against control limits established in each area. Those critical processes are 

listed below. 

A.2.2.1 Cost Estimating Relationship (CER) 

CERs are used within proposals for elements that are estimated in relationship to another element 
i e ODC as a percent of labor dollars. CERs will be monitored to ensure that the actuals incurred 
are within the control limits established in the CER methodology and therefore remain valid CERs. 

A.2.2.2 Basis of Estimate (BOE) 

A BOE contains the estimating methodology for a proposed task including^ 
the logic used, i.e., the historical costs referenced and justification for differences. The results ot 
the joint proposal review checklist questions regarding BOEs will be monitored against the control 
limits established as quality measures. 

A.2.2.3 Material 

Material includes electrical, mechanical, and engineering purchased parts and raw material used in 
the fabrication of hardware. The results of the joint proposal review checklist questions regar ing 
material will be monitored against the control limits established as quality measures. 

A.2.2.4 Subcontract Analysis 

^iihcnntract analysis is performed on all major subcontracted products or services that are in excess 
of $500 000 to determine that the price is fair and reasonable. The results of the joint proposal 
review ch^klist questions regarding subcontract analysis will be monitored against the control 
limits established as quality measures. 

A.2.2.5 Assist Field Pricing Audit Reports 

Assist field pricing is performed by DCAA when a prospective subcontractor denies TRW access 
totheir books and^ecords. The average number of days to obtain completed Assist Field Pricing 
Audit Reports will be monitored against established control limits. 
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A.2.2.6 Other Direct Cost (ODC) 

ODCs include accounts for items used in support of program activities i.e. travel renrn and 
computing. The results of the joint proposal review checklist questions regarding material ’will he 
monitored agamst the control limits established as quality measure. 8 8 al Wl11 be 

A.2.2.7 Direct Labor Rates 

Direct labor rates will be tracked and compared to the negotiated rates on a comDosite basis As 
TOw“nhfDPRoTtSf W,,hfa U ’ e ' StabUshed tolera ”“ s ' » additional effort is^quired by either 

A. 2.2.8 Indirect Rates 

• be l ? c )“ <i and f° m par«d to the negotiated rates for the major indirect cost pools 
? fs S - raUs remain wilhin established parameters, to additffl 

‘fjl ‘i^V^d by either TRW or the DPRO staff. Should the rates fall outside YhetoleranS 
7^^ wt provide additional suppon/explanation in order to assist the DPRO in their 
determmation as to whether an adjustment to billing is appropriate. 

A.2.3 Procurement (Subcontracts and Purchasing) 

Four critical processes have been jointly identified by the DPRO and TRW The monitoring of 

TOW P S°^haol b f, f r liU,ed b / ^ 5a, a displayed on SPC ch^^Sby 

■tw' ? P ? Ch ^ rts ] ^ vi11 be . Prepared to identify both "out of control" as well as normal operations 
The out of control condition if any, will be clear and indisputable and the results of any 
corrective action will be readily visible. The four critical processes area are shown below. 

A.2.3. 1 Advance Notice and Prior Consent 

Customer advance notification or prior consent (or both) may be required from TRW customers 

fCra-lShT^ 8 3 P roc f TRW Consolidated Prime Contract Summary 

tS PW, StS tho f procurements which require advance notification or prior consent (or both) ™ 

S— r ySB “ d Revi ® w ( PAR ) wiI1 determine, if required, that the prooirement 
package has the advance notice or prior consent (or both) and that it was done in compliance with 
the applicable procedures. If there is an omission it will be noted for internal improvement 

“ d mC ! ud , ed !?> SPC chan for ,ha ‘ m ° mh - ** “section will K 
P , § e approval. In addition, an annual random sample of all procurement packages will be 
conducted to ensure all completed packages have satisfied the requirement foran advance notice or 
prior consent (or both). The number of discrepancies found by the PAR will determine if adequate 
instructions or additional training are necessary. ** 

A.2.3. 2 Price and Cost Analysis 

Proposals and quotations obtained in support of potential POs/SCs (or changes thereto) require 
certified cost or pricing data whenever the proposal exceeds the dollar threshold established by 

S £ ° r wb t never 11 !■ reqUKed hy terms ^ conditions of the TRW customer contract 
and do not meet the exceptions to the requirements for certified cost or pricing data. The PAR will 

Jtc h!™!. 6 ’ ^ squired, that the procurement package has the required price or cost data and that it 
wH? h? 1 “ P ll . ance ,'y ith ^ applicable procedures. If there is an omission or discrepancy it 
will be noted for internal improvement purposes and included in the SPC chart for that month 

^ rTeCtl ° n f W1 \ be mad< r,P[ ior t0 ? ackage a PP ro val. In addition, an annual random sample of 
all procurement packages will be conducted to ensure all completed packages have satisfied the 
requirement for cost and pricing data. The number of omissions or discrepancies found by the 
r AK will determine if adequate instructions or additional training are necessary. 
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A.2.3.3 Small/SmaU Disadvantaged Business Concerns 

Quarterly reports are submitted to the DPRO concerning TRWs achievements on meeting goals 
established for Small/Small Disadvantaged Business Concerns (S/SDBCs), Historically Black 
Colleges and Universities (HBCU), and Minority Institutions (MI). 

A.2.3.4 Certification 

Certifications are required by public law and the Defense Acquisition Regulation/Federal 
Acquisition Regulation (DAJUFAR) under specified conditions in TRW procurement packages. 
These certifications involve such things as Equal Opportunity in Employment, the Clean Air Act, 
The Federal Water Pollution Control Act, and related regulations of the Environmental Protection 
Agency (EPA). The PAR will determine if the required certifications are included and correctly 
filled out. If there is an omission or discrepancy it will be noted and included in the SPC chart for 
that month. Any correction will be made prior to package approval. In addition, an annual random 
sample of all procurement packages will be conducted to ensure all required certifications are 
included and correctly filled out. the number of omissions or discrepancies found by the PAR will 
determine if adequate instructions or additional training are necessary. 

A.2.4 Property 

The Government Property Administrator is required to perform an annual survey that reviews 
TOW's compliance with its property control procedures. The government's guidelines for 
conducting the reviews were used. Under the FAR provisions if several property system criteria 
are met, the annual survey requirement can be changed to a biennial audit requirement. Therefore 
to meet the FAR criteria for biennial survey review, the government audit guidance for reviewing 
the TOW property control system was used to select jointly seven processes which are essential to 
the DPRO oversight responsibilities. Those seven processes are shown below. 

A.2.4. 1 Acquisition of Equipment 

The acquisition of special test equipment requires government approval in advance of the 
acquisition unless identified in the contract. This evaluation criteria will measure the acquisition 
for equipment and the compliance with the advance notice and approval request to the government. 
Purchase requisitions will be reviewed for inclusion of end use codes, the proper cost account 
codes, and the advance government approval to acquire. The matrices will be purchase 
requisitions reviewed and with correctly completed criteria. 

A. 2 . 4 . 2 Movement of Equipment 

Location of equipment is an important aspect of inventory control of assets. The Automated 
Property Movement Authorization (APMA) is a document created in the records computer database 
which provides for the hard copy movement document and electronic location changes in the 
records. This evaluation criteria will evaluate the timeliness of the closure of the movement 
documents and location updates for equipment moves. The APMAs will be reviewed for the 
timeliness to complete (close) the APMA; this action updates the location changes of equipment. 
The matrices will be the number of open APMAs exceeding "x" days for each group. 
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A.2.4.3 Physical Inventory 

Physical inventories of material not only fulfill a contract requirement, but also is important to 
ensure that parts and assemblies received into the storerooms remain available to meet the 
production schedule and contract delivery requirements. This evaluation criteria will measure the 
annual schedule and the actual performance against that schedule. The material physical 
inventories are completed annually and performed throughout the year. The matrices will be 1) a 
measure of actual performance against the planned schedule, and 2) a measure of the variances 
between the records balance-on-hand and the actual count of each item. 

The physical inventory of equipment can be extended to longer pe riods than material b ecause of 
several reasons. Equipment has a longer life span. There is less turnover of equipment than 
material, and material is purchased closer to its usage date. However, it is still important to verify 
that equipment items are available. The evaluation criteria will measure the physical inventory 
schedule and performance to that schedule. The physical inventory of equipment is on a biennial 
basis occurring in the odd number years. The matrices will be 1) the measure of the actual 
performance against the planned schedule, and 2) a measure of the variances between the records 
and the actual count of the equipment items on a contract basis. 

A.2.4.4 Use of Equipment 

Equipment is acquired to support a development or production task or a test requirement under 
contract. These equipment items are retained do to their use; and when no longer required, they 
become excess to that contract. When excess, the items should be legalized on another contract or 
disposed of. This evaluation criteria will evaluate current and projected use for the items. As a 
result of a TQM/CPI effort, all equipment will be coded for its utilization and retention 
requirements. The matrices will be timeliness in completing this semiannual task and quantity of 
equipment in each of the six retention code categories. 

A.2.4.5 Equipment Maintenance 

Government equipment must be properly maintained to provide for maximum life and correct 
performance. The criteria for maintenance involve both the determination of need for maintenance 
and, where appropriate, the performance of maintenance. Maintenance may include any of the 
following: inspection (evaluation), operator performed (functional), electronic or mechanical repair 
(corrective), or routine periodic (preventative). Based on the application of the equipment, 
calibration may be required. The custodian may elect to have designated equipment maintained by 
virtue of calibration being performed by a centralized calibration organization. The evaluation 
criteria for maintenance will measure the actual performed maintenance against the custodian's 
determination of need for said maintenance. 

A.2.4.6 Contract Property Closure 

The process to ensure proper disposal of all government property accountable to a contract is 
generally one of the last tasks to be completed prior to contract closu re. This evaluation criteria 
will evaluate the completion of all tasks required to Issue a property closeout certification. The 
process ofproperty contract closure is an extended task that can often delay the contract closeout 
activity. Tnis process requires focused attention and reporting. The matrices will be the time 
period from initiation of contract closeout until the property certification is issued. 
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B. Engineering and Manufacturing 


B.l Critical Processes 

These defined critical process are still generic in nature and each critical process will be further 
defined by product line or cost center code. 

B . 1 . 1 Systems Engineering 

B.l. 1.1 Concept Design 

Determines generic system elements and die manner in which those elements will satisfy functtonal 
requirements. 

B. 1 . 1 .2 Requirements Allocation 

Defines the performance requirements needed for each system element to attain the overall 
performance requirements of the system. 


B.l. 2 Detailed Design 

B 1.2.1 Unit Specification Generation 

Defines discrete unit level requirements, derived from the mini specification flow down from 
systems engineering. 

B. 1.2.2 Design 

Determines the manner in which the unit will be designed to satisfy the unit specification 
engineering and M&P. 

B.l. 2.3 Engineering Drawing 

Transfers conceptual drawings and schematics to detailed drawings used to build the design. 
B. 1 .3 Parts Procurement 
B. 1.3.1 Specification Generation 

Takes the parts identified from design and program quality requirements and determines the 
specifications needed for said parts. 

B. 1.3.2 Purchase Order (PR to PO) 

Purchases a specified part from a list of possible vendors. 

B.l. 3. 3 Supplier 

The vendor builds the part to specification. 


B.l. 3.4 Receipt to Issue 

Receiving of the parts and parts data, 
parts to their respective customers. 


performing V&H testing if required, and delivering those 
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B. 1 .4 Subcontracts 


“■?* process is vc fy simiIaj ' t0 parts procurement process in that two of its three 
that usually requires some developmental work by the subcontractor. 


B. 1 .5 Manufacturing 
B. 1.5.1 Fabrication 


Transforms raw materials into some desired output. 
B. 1.5.2 Assembly 

Assembles piece parts into some desirable unit. 

B. 1.5.3 Test 


To test either functionally, visually, or dimensionally, the fabricated 
if it meets specifications. 


or assembled part to determine 


B . 1 . 6 Support Equipment 

TTie support equipment process is a microcosm of the top-level processes. The critical processes 

deCtriCa ! design> produCt des ^’ fabrication and assembly and 
integration and test. The mini-spec generation process takes an equipment spec and test * 

requirements and generates a test set mini-spec. The electrical desigS process es the test set 
mini-spec and identifies parts and new processes, schematics, conceptual drawings interface 
an A ra r k t ay ^ UtS; • Fhe prod “ ct desi S n Process takes the schematics, identified 

,S h and m K, ni * SpeC 10(1 transfon ns those into a completed engineering drawing 
TTie fabrication and assembly process manufactures the subassemblies and assemblies Finallv the 

equffm. 1651 Pr0Ce$S intCgrateS ^ ,CStS th6 SubaSSCmblies a com^ted piece of support 


B. 1 .7 Systems Integration and Test 

Assembly, integration, and test of various boxes into a satellite ready for launch. 

B. 1.7.1 Requirements Definition 

Transforming systems engineering requirements into a test requirements document. 

B . 1 . 7 . 2 Procedure Definition 

Taking the requirements documents and developing a detailed assembly and test procedure. 
B. 1.7.3 Assembly 

Takes hardware, software, and detailed test procedures and assembles a satellite. 

B.l.7.4 Integration 

Takes the assembled systems and tests it to determine if requirements are being met. 
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B. 1.7.5 Acceptance Test 

The process of testing a fully assembled and integrated satellite to determine if the system is ready 
for launch. 


B.2 Metrics Identification 

Critical Processes 
Systems Engineering 


Detailed Design 


Subcontracts 


Parts Acquisition 
Support Equipment 


Systems Integration and Test 
Manufacturing 


Metrics , 

• Time between program start and release of unit 

specs 

• Actual time required to release unit specs vs. 
planned time 

• Number of engineering changes per drawing 

• Deviation of total design hours from standard 

• Number of first pass successes in design 

• % of letter subcontracts 

• Average days to definitize 

• % of subcontracts with late deliveries 

• Delivery schedule of subcontracts 

• Integration rejects as a % of subcontracts 

• Nonconforming material as a % of subcontracts 

• Preventable notices as a % of subcontracts 

• Number of notice of delay of material per lot 

• Number of procurement deficiency reports per lot 

• Material acquisition time in weeks 

• Design hours per released drawing 

• Fabrication and assembly hours per unit 

• Design hours per gate 

• Design hours per unit 

• Number of deliverable source instructions 
developed per hour 

• Design errors per released drawing 

• Avoidable rework per assembly hour 

• Integration returns 

• Time lost vs. time worked 

• Engineering caused rework as a % of bench hours 
for fabrication/assembly and test 

• Manufacturing caused rework 

• Other types of rework 

• Number of nonconforming materials 

• Number of test discrepancy reports 
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C. Quality Assurance 

The authority and responsibility for managing the contractor's quality program should be complete 
properly assigned, and documented using written policies, procedures and work instructions The' 
following major support processes (MSPs) should be addressed as part of the contractor's quality 
system. n 3 

C. 1 QA Internal Audits/Training and Quality Cost (QA-1) 

The quality organization must: 

• Assign authority and responsibilities to organizations performing quality functions. 

• Establish a training and certification program. 

• Have an internal audit system. 

• Have provisions for collection and use of quality cost data. 

C.2 QA Planning/Work Instructions (QA-2) 

The contractor's planning program should provide for timely and effective planning, which has: 

• Methods for accomplishment of comprehensive contract reviews. 

• Provisions for inspection and test planning during the earliest practical phase of contract 
performance. 

• Methods for verifying that inspection and test planning is compatible with manufacturing 
methods, processes, drawing requirements and inspection instructions. 

C.3 Records (QA-3) 

The contractor's quality system is to assure that records are kept current, complete, legible, and 
accurate during all phases of design, development, manufacturing, test, etc., as a means of 
maintaining the continuous history of the product/service, e.g., fabrication and assembly history; 
build-up and disassembly, repairs, rework, results, configuration data, etc. 

C.4 Software QA (QA-4) 

This MSP will be assessed by Software Process Improvement Initiative (SPII). 

C.5 Nonconforming Supplies (QA-5) (TRW, Suppliers, and Subcontractors) 

The Quality organization must assure: 

• Effectiveness authorized PR/MRB dispositions. 

• Timeliness of dispositions. 

• Provide methods for the positive identification, segregation and storage of nonconforming 
supplies in adequate holding areas. 

C.6 Corrective Action (QA-6) (TRW and Suppliers) 

The contractor's corrective action program should provide for the timely detection of discrepancies 
and ensure timely and positive action is taken to eliminate the cause of defects. The corrective 
action system should include requirements for: 

• Effectiveness of completed corrective action (recurrence). 

• Timeliness of corrective actions. 

• Effectiveness of Supplier and Subcontractor Corrective Action(s). 
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C.7 Supplier QA (QA*7) (Suppliers and Subcontractors) 

The contractor's quality program should ensure that all supplies and services purchased from 
suppliers conform to contractual requirements. The effectiveness and integrity of quality control 
and corrective action by suppliers should be assessed and reviewed by the contractor. The 
contractor's quality program should: 

• Ensure that products are inspected on receipt to determine acceptability. 

• Have objective evidence data to be used for adjusting the extent of receiving inspections. 

• Provide for an effective vendor rating system. 

• Ensure that untested product and raw material are identified and segregated from those tested 
and accepted. 

• Provide for functional tests to be performed to the required specification, technical order, 
drawing or contract, if testing is required. 

• Ensure that suppliers' quality systems meet the quality requirements of the 
subcontract/purchase order (flowdown of QA requirements). 

C.8 Metrology, Calibration, and Tooling (QA-8) 

The contractor's quality program should provide for an effective metrology and calibration system, 
for standards and measuring test equipment. The contractor's quality program should: 

• Ensure that required certified measurement standards as well as gauges, testing, and 
measurement equipment are available and used. 

• Ensure testing and measuring equipment, including personally-owned tools when 
authorized, to be recalibrated on a regular basis to determine that they are of required 
accuracy. 

• Maintain records for the control of calibration activities. 

• Ensure that calibrated measuring and test equipment have evidence of traceability either 
through primary or reference standards to the National Institute of Standards and 
Technology or natural physical constants. 

• Ensure that environmentally controlled areas are maintained. 

• Ensure that when measuring and test equipment becomes damaged or inaccurate they be 
effectively controlled, replaced, or repaired. 

• Ensure that tooling used as a media of inspection is calibrated or certified, and proven for 
accuracy before use and reinspected at established intervals which ensures the adjustment, 
replacement, or repair of the tooling which becomes inaccurate. 

C.9 Materials, Treatments, and Processes (QA-9) 

The contractor's quality program should provide for monitoring materials, treatments, and 
processes such as soldering, welding, heat treating, etching, plating, and promptly correcting 
improper process monitoring methods or inspection and test techniques. This includes. 

• Adequacy of materials storage and usage and proper nondestructive inspection controls. 

• Contractor personnel should monitor special controls for age sensitive items such as 
chemicals, rubber lubricants, paints, and adhesives. 

• Material should be protected against deterioration, damage, contamination, or electrostatic 
discharge damage in use or in storage. 

• The contractor should have provisions for assuring the control of processing environment, 
as well as the necessary degree of certification, inspection, authorization on and monitoring, 
for such specialized processes. 
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C. 10 Inspection and Test (QA-10) 

The contractor's quality program should ensure final inspection and test of products are performed 
by quality personnel. Such inspection and testing will provide a measure of the overall quality of 
the completed product. Testing will be performed so that it stimulates to a sufficient degree 
product end use function. * 


D. Software 


D.l Software Process Capability 

The ability of the contractor to establish, maintain, and improve a software development 
infrastructure capable of fulfilling contractual cost, schedule, and technical constraints. 

D . 1 . 1 SEI CMM-Based Evaluations 

D . 1 . 1 . 1 Assessment Results 
D. 1 . 1 .2 Evaluation Results 

D.l. 2 Technology Improvements 

D.l. 3 Technology Dissemination 

D. 1 .4 TQM/CPI Activities 

D.2 Software Process Management 

The ability of contractor management to development and implement a plan to produce quality 
software within contractual cost, schedule, and technical constraints. 

D.2.1 Management Oversight 

D.2. 1.1 Cost Variance 
D.2. 1.2 Schedule Variance 

D.2.2 Project Metrics 

D.2.2.1 Software Development Status 
D.2.2. 2 Software Capabilities Integrated 
D.2. 2.3 Requirements Verification 
D.2.2.4 Software Problem Reports 

D.2.3 Customer Assessments 

D.2. 3.1 Award Fees 
D.2.3.2 Other Assessments 

D.2.4 Milestone Results 

D.2.5 CDRL Submittal 

D.3 Software Process Performance 

The evaluation of the quality and the cost, schedule, and technical impacts of the contractor's 


software 

development process. 

D.3.1 

IQUE Reviews 

D.3.2 

TRW Compliance Reviews 

D.3.3 

Customer Compliance Reviews 
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